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Tr ainin g Evaluation With 360-Degree Feedback 

Froukje A. Jellema 

University of Twente, The Netherlands 

Training evaluation is an important issue in many organizations. Even though organizations are becoming 
more interested to measure the effects of their training investments, organizations often lack the tools for 
the measurement of training effects, especially at the level of changed behavior . This paper will focus on 
the use of 360-degree feedback for evaluation of changed behavior as a result of training. The results of an 
experimental study in a large Dutch hospital will be described and discussed. 

Keywords: Training Evaluation, 360-degree Feedback, Behavioral Change 

Human resource development (HRD) is a process of developing and unleashing human expertise through 
organization development and personnel training and development, for the purpose of improving performance 
(Swanson, 2000 AHRD). Training evaluation can be useful to find out whether investments in training are 
contributing to performance imp rovement. One evaluation focus is changed behavior as a result of training 
(Kirkpatrick, 1994). Though change in behavior is in most cases not an end in itself, it is a necessary intermediary 
step to evaluate whether the training contributes to performance improvement. That is why in this paper the focus is 
on the evaluation of behavioral change. 

Evaluation of changed behavior is done for only 1 1 percent of the training programs, though companies expect 
to increase this proportion in the future (1999 ASTD report). A main reason why training evaluation is not often 
focused on behavioral change of trainees, is the lack of appropriate methods. The specific aim of this paper is to 
examine one particular method that may be useful in this context. This method is 360-degree feedback, where 
employees receive feedback about behavior from a ‘full circle’ of co-workers such as the supervisor, peers, 
subordinates and clients. 36(Wegree feedback may be used to determine whether co-workers have experienced a 
difference in the behavior of the trainee. The main research question of this paper is whether 360-degree feedback is 
indeed useful for training evaluation. 

Theoretical Framework 

Though the use of 360-degree feedback for training evaluation is often mentioned in evaluation handbooks (see for 
example Brown and Seidner, 1998; Kirkpatrick, 1994; Robinson and Robinson, 1989), this method has rarely been 
studied in an evaluation context, though often in an employee developmental context. 

One study in which 360-degree feedback is used to evaluate a training program, is the study by McLean, bytsma, 
and Kerwin-Ryberg (1995). This study included 73 ratees, who received 360-degree feedback both before and two 
years after a training program. Though participants’ skills increased during these two years, it was difficult to 
attribute this behavioral change to the training program, due to the long period between pre- and posttest and the fact 
that a control group was not included. The authors 

concluded that 360-degree feedback should be applied very cautiously as a tool to evaluate training. 

Another study in which 360-degree feedback is used for training evaluation, and in which a design with a 
control group was ’used, is the study of Van Sandick and Schaap-Neuteboom (1993). This study was situated in a 
large Dutch supermarket chain and focused on a training program for department-managers. This study included 162 
ratees distributed over two training groups and one control group. In this study a training effect was measured (i.e. 
0.2 on a 5-point scale). However, this study was limited by the fact that only one supervisor and one peer were 

involved in the feedback. _ , t . . , . . , 

Another example is the study of Rosti and Shipper (1998), where 360degree feedback is used to study the 
impact of training in a management development program based on 360-degree feedback. Feedback was collected 
both before and after the program to enhance learning and evaluate training. Results, when analyzed by means of a 
series of four matched-pairs tests, were supportive that the change in the experimental group was significantly 
different than the change for the control group, and in the expected direction. 

Since organizations are becoming more interested in measuring behavioral change as a result of training, it is 
interesting to study the usefulness of 360-degree feedback for this purpose. Since previous studies in this area are 
rare and often limited, it was decided to use as strict an experimental design as possible to again look at the 
usefulness of 360-degree feedback for training evaluation. 

Copyright © 2002 Froukje A. Jellema 
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Research Questions 

The main research question of this paper is whether 36(kdegree feedback can be useful for training evaluation. More 
specifically, the questions are: 

1. Is training evaluation by means of 360 -degree feedback reliable? 

2. Is training evaluation by means of 360-degree feedback valid? 

3. What training effects are found with 360-degree feedback? 

Methodology and/or Research Design with Limitations 

The methodology that is used to answer the questions, is a qua si-experimental study. This study is situated in a large 
Dutch hospital. Recently, this hospital has created a new professional job for nurses, the senior nurse, hierarchically 
situated below the head nurse. All senior nurses have to attend the course, that has been specifically developed for 
this function. This is an extensive course, consisting of 18 outdoor training days during a nine-month period. 
Coaching skills are an important part of the program. 

At the time of this study, about 40 senior nurses were working at the hospital. Though all senior nurses have to 
participate in the program, only 14 nurses could attend at the same time. That is why it was possible to study three 
conditions. Group 1 had already started with the program at the beginning of this study and received only a 360- 
degree feedback post-test. Group 2 received a pretest with 360-degree feedback before attending the program, mid 
afterwards a posttest with 360-degree feedback. Group 3 received the feedback at the same time as group 2, but did 
not attend the program. All post-tests took place after ten training days, about seven months after the pretest. It 
would have been preferred if the post-measurement had taken place at the close of the training program, but that 
excluded the use of a control group, since group 3 would have started with the training by then. 

The Dutch 360 -degree feedback instrument Reflector was used in this study, consisting of eight behavioral 
competencies: dealing with stress, integrity, active listening, creating partnership, diagnostic skills, stimulation, 
giving feedback, and adapting. Each behavioral competency consists of five items, so the questionnaire included 40 
items. The response scales that are used in Reflector are 5-point scales were raters select a position between opposite 
statements of behavior. The reliability and validity of Reflector has been examined in previous studies (see Jellema, 

Visscher and Mulder, 2000) and was found to be satisfactory. 

Each senior nurse was allowed to select his or her own raters, but preferably their supervisor, two peers and four 
subordinates Clients (in this case: patients) were not included. The ratees were encouraged to ask the same raters to 
provide feedback at the pre- and posttest, for as far as this was possible. All ratings could by given anonymously. 
After the results were analyzed, the results were summarized in reports and send to the home addresses of the ratees. 

All ratees were invited to join in a follow-up session. , 

The post-tests included a retrospective self-test and a questionnaire with questions regarding background, 
motivation for the training program, perceived support on-the-job, etc. Out of group 2 and 3, eight senior nurses 
were found willing to participate in a coaching simulation, that took place at the time of the post-test. Main purpose 
was to study correlation between 36(kdegree feedback (post-test) scores and scores of the simulation and thus to 
gain insight into the validity of 360-degree feedback. The research design is summarized in Table 1 . 

Table 1 ■ The research design of this study 

360° & retrospective self & 
background 

training 360°& retrospective self & background & 

coaching simulation 

no training 360°& retrospective self & background & 

coaching simulation 



Group 1 


training 


Group 2 


360° 


Group 3 


360° 



Response 

At the start of the study each group consisted of 14 senior nurses. However, several senior nurses could not partake 
because they were ill (especially in group 3, the group that was on the waiting list for the program), left the 
organization or did not want to cooperate. At the posttest, again several nurses were ill and could not participate. 
Table 2 summarizes the response. 
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Table 2. Response of the evaluation at the OLVG 





Pre-test 


Post-test 


Post-test 






360 


360 


Background & 
Retrospective 


360 


Background & 
Retrospective 


Simulation 


Group 1 




n« 13 
(95 raters) 


n= 13 








Group 2 


n=12 

(86 raters) 






n= 12 
(78 raters) 


n= 12 


n-4 


Group 3 


n = 9 

(69 raters) 






n = 7 

(48 raters) 


n= 7 


n-4 



In this paper, the focus will be on group 2 and 3, since these groups received a pre- as well as a posttest. The 
pretest included 21 senior nurses, 12 in the experimental group and nine in the control group. A total of 155 raters 
were involved, an average of 7.4 raters for each ratee. Raters were divided as follows: self (21), supervisors (20), 
peers (62) and subordinates (52). At the posttest, a total of 18 senior nurses again participated in the 360-degree 
feedback. One senior (in group 2) participated in the posttest but had not participated in the pretest. A total of 126 
raters were involved, an average of 6.6 raters for each ratee. Raters were divided as follows: self (19), supervisors 
(17), peers (50) and subordinates (40). About 35% of the seniors had asked exactly die same raters as at the pretest. 
In the other cases, one or more of the raters at the posttest had not been included in the pretest. All senior nurses 
filled in the questionnaire with background questions and the retrospective questionnaire. 

Since this study was a quasi-experimental study, and the groups were not selected at random, they were 
compared on several variables to find out if they were different (see Table 3). 



1 BUIV -F- * ™ | 

Variable 


Group 2 (n=12) 


Group 3 (n- 9) 


Gender 


53.8% female; 46.2% male 


77.8% female; 22.2% male 


Age 


36 (4.6) 


34(8.1) 


Working experience 


12.2 (5.4) 


12.7(9.6) 


Experience senior nurse 


.8(1.3) 


2(1.8) 


Experience coaching 


66.7% 


88.9% 


Perceived need to develop coaching skills 


8.3% very high; 75% high 


11.1% very high; 66.7% high 


Motivation for program 


25% very high; 66.7% high 


22.2% very high; 66.7% high 



As can be seen in Table 3, the groups are different on many of these variables. However, differences between groups 
were tested by means of chi-square (gender and experience with coaching), ANOVA (age and need to develop 
coaching skills) and Kruskal Wallis (experience and motivation), and none of the differences are significant. This is 
probably due to the limited number of respondents that was involved in this study. 

This limited number of respondents and the fact that respondents were not selected at random for each training 
group, is the main limitation of this study. Interpretation of results may be difficult due to this limitation. In addition, 
the ratees that were included at the pretest are not always (though often) die same individuals as the ratees at the 
posttest. In addition to this, it is a limitation of this study that it was not possible to train the raters in advance. 



Results 

In this section the following results will be presented: the psychometric properties of the 360-degree feedback 
instrument and the measurement of behavioral change as a result of the training. 



Psychometric Properties of the 360-degree F eedback Instrument 

In this section the focus will be on the internal consistency and inter-rater reliability and validity of the 
instrument. 

Internal Consistency. Internal consistency is measured by means of reliability analysis of the scales. The 
analysis is done on the complete data set (pre - and posttest, including group 1). Table 4 summarizes the alpha 

coefficients. 
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Table 4. Internal consistency of the scales 



Competency 


Alpha 


Active listening 


.60 (n=357), .68 after deleting item 2 


Creatine partnership 


.66 (n=341) 


Diagnostic skills 


.76 (n=333) 


Stimulation 


.70 (n=336) 


Giving feedback 


.74 (n=»353) _ 


Adapting 


.81 (n=326) 


Integrity (control competency) 


.72 (n=356) 


Dealing with stress (control competency) 


.78 (n-349) 



Two scales are below .7, i.e. active listening and creating partnership. The alpha of active listening can be increased 
to .68 by removing one item. The alpha of creating partnership and the other behavioral competencies can not be 
increased by removing items. When all items are considered as one scale (‘coaching skills’), the alpha is .94 (n 272) 



Inter-rater Reliability 

Since most of the senior nurses have received feedback from more than one peer and/or subordinate, it is 
interesting to see if scores within each rater-source are more or less the same. Since many ratees have feedback from 
three or more peers and/or subordinates, it was not possible to use a correlation coefficient. That is why the standard 
deviation is used as a measure for within-source agreement. The data was first analyzed separately for each ratee 
after which an average score for all ratees was computed. The standard deviation is computed for all competencies 
and for an overall score of all competencies. Table 5 shows the average standard deviation for peers and 
subordinates. 



Table 5. Within-source standard deviation (or peers and subordinates 



Competency 


Peers 


Subordinates 


Dealing with stress 


0.48 (n=44) 


0.41 (n=39) 


Integrity 


0.33 (n=44) 


0.39 (n=38) 


Active listening 


0.44 (n=44) 


0.43 (n=40) 


Creating partnership 


0.48 (n=44) 


0.49 (n=39) 


Stimulation 


0.43 (n=44) 


0.48 (n=38) 


Diagnostic skills 


0.52 (n=43) 


0.49 (n=38) 


Adapting 


0.48 (n=44) 


0.59 (n=39) 


Giving feedback 


0.43 (n=44 ) 


0.44 (n=39) 


Overall score 


0.45 (n=44) 


0.46 (n=40) 



The average overall standard deviation for peers is .45. This differs slightly for each behavioral competency 
with most agreement between peers on integrity and less agreement on diagnostic skills. The average overall 
standard deviation for subordinates is .46, which again differs slightly for each behavioral competency. Most 
agreement between subordinates is on integrity and less agreement on adapting. . . 

Since the main characteristic of 360-degree feedback is that several different rater sources are included, it is 
interesting to study agreement between rater-sources. Whenever a ratee has two or more peers and/or subordinates, 
an average score for these raters is computed. This average peer-score and average subordinate-score is used in the 
analysis. The analysis is based on the complete data set (both pre- and posttest, including group 1). For each ratee, a 
score was computed for self, supervisor, peer and subordinate. These scores were averaged for all ratees. In Table 6, 
the average scores are summarized. 



auie O. si vcru£z: otw e 
Competency 


Self (n=53) 


SV (n=50) 


P(n=52) 


SO (n-50) 


Dealing with stress 


4.03 (.61) 


4.00 (.75) 


3.96 (.55) 


4.10 (.52) 


Integrity 


4.51 (.38) 


4.57 (.38) 


4.48 (.33) 


4.44 (.51) 


Active listening 


4.01 (.44) 


3.87 (.60) 


3.90 (.38) 


3.99 (.48) 


Creating partnership 


3.96 (.42) 


3.96 (.67) 


3.92 (.43) 


3.90 (.58) 


Stimulation 


3.96 (.46) 


4.01 (.64) 


3.97 (.40) 


4.04 (.43) 


Diagnostic skills 


4.00 (.43) 


4.00 (.57) 


4.05 (.41) 


4.08 (.52) (n=49) 




3.97 (.57) (n=52) _ 


3.87 (.74) 


3.94 (.50) 


3.98 (.58) 


Giving feedback 


4.08 (.45) 


4.06 (.60) 


4.15 (.41) 


4.22 (.44) 


Overall score 


4.06 (.35) 


4.04 (.54) 


4.05 (.33) 


4.09 (.45) 
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As can bee seen, the overall scores of all rater-sources are very similar. The standard deviation in Table 6 
indicates the differences between ratees. When the assumption is that ratees have different levels of performance, 

this coefficient should be large; it is, however, rather low. nn .. 

Correlation between these sources was studied by means of Ttests, since the scores appear to be normally 
distributed. In Table 7 the coefficients are summarized. The marked coefficients are significant (at .95 level). 



aDie /. correlation ueiwt 
Competency 


xn r uicr ji/k 

s-sv 

(n=50) 


S*P 

(n=52) 


s-so 

(n=50) 


SV-P 

(n=49) 


SV-SO 
(n=47) 


P-SO 

(n=49) 


Dealing with stress 


0.39* 


0.20 


0.08 


0.44* 


0.30 


0.31* 




0.11 


-0.04 


0.05 


0.15 


0.27 


0.00 




0.27 


-0.03 


0.09 


0.07 


0.12 


0.04 


Creating partnership 


0.22 


020 


-0.13 


0.17 


0.26 


0.06 


Stimulation 


0.18 


-0.02 


0.15 


0.12 


0.30* 


0.07 


Diagnostic skills 


0.27 


0.05 


0.02 


0.23 


0.24 


-0.16 




0.35* 


023 


0.11 


0.41* 


0.29* 


0.01 


Giving feedback 


0.32* 


0.03 


-0.02 


0.28 


0.16 


-0.01 


Overall score 


0.34* 


0.00 


0.03 


0.29* 


0.32* 


0.03 



The fact that correlation appears to be moderate for most competencies and rater-sources, implicates that scores 
should by presented for each rater-source separately. This is in accordance with what you might expect for 360- 
degree feedback. Correlation between peers -subordinates, peers-self and self-subordinates, is in some cases even 
negative. When the correlation of the overall score is used as a measure of rater-correlation, the correlation between 
supervisor-self, supervisor-peer, and supervisor-subordinates is significant. These correlation coefficients are in 
agreement with the low correlation that is often found in other studies (see for example Hams and Schaubroeck, 
1988) However, correlation between self-peer, self-subordinate and peer-subordinate are much lower than have 
been found in previous studies. Interesting to note is that the correlation between the supervisor and any of the other 
rater sources is significant, while correlation between the other sources is not significant. 



Validity 

To study the validity, the 360-degree feedback scores were compared to another, more objective, measure of the 
same behavior. Eight senior nurses were found willing to participate in a coaching simulation that consisted of a 
coaching conversation that the senior nurse had with an employee, played by an actress. The simulation focused on 
some of the competencies that had been included in the 360-degree feedback i.e. active 1,sten ' n 6- c ^ eat '" g 
partnership, diagnostic skills and stimulation. The eight conversations were recorded on videotape and two masters 
students evaluated the videotapes by means of the same questionnaire that was used for the 3 60 -degree feedba ^ ' 
The students were trained in advance (with an extra videotape) to use the instrument and assess the coaching 
behavior Each student rated each ratee, after which they discussed the results and tried to come to agreement. Since 
each student-score correlated significantly with the agreement-score, this agreement-score is used in the analysis. 
Because the simulation was in the same period as the 36<Megree feedback posttest, the posttest scores are used. 
Table 8 shows the correlation that was found. 



Table 8 Correlation be tween 360-degree feedback and coa ching simulation 



aoie o. correlation ueiw e 


Sim-360 Self 
(n=8) 


Sim-360 SV 
(n=8) 


Sim-360 peers 
(n=8) 


Sim-360 SO 
(n=8) 


Active listening 


-.16 


-.58 


.31 


-.70 


Creating partnership 


.17 


-.33 


.25 


-.31 


Diagnostic skills 


-.25 


-.72* 


.40 


-.44 


Stimulation 


-.44 


-.60 


.29 


-.58 



SV=supervisor; SO = subordinates 



As can be seen, the correlation between the 360-degree feedback and simulation is in most cases negative. Only 
peers have a positive (though not significant) correlation with the simulation-score. 



Measuring Behavioral Change as a Result of Training 

in this section the focus is on the gain scores between pre- and posttest. First, it is important to focus more 
specifically on the self-scores, since these may be sensitive to ‘response-shift bias’: because ratees in group 
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receive intensive training of coaching skills, their ideas about coaching may change. Tins may result in the fact that 
self-scores at the posttest are based on another perspective of coaching, which makes them difficult to compare with 
the pretest scores. Self-scores could be lower at the posttest, even though performance has improved, because 
trainees have learned how difficult coaching actually b and are more demanding of their own performance^ The 
same process may happen for ratees in group 3, who received no training but who did receive a feedback-reportafter 
the pretest, joined in a follow-up meeting and were encouraged to make a personal development plan. These 
individuals may also have changed their ideas about coaching. That is why a retrospective self-test was me hided 
with the posttest, where ratees were asked to again rate their coaching skills at the time of the pretest. Correlation 
between pretest self-scores and retrospective scores was studied by means of a T-test (see Table 9). 



autt 7. Ul/nCHmun rrww,. ~ v 

Competency 


Pretest self-score 


Retrospective self score 


Correlation 


Dealing with stress (n=3D 


4.01 


3.81 


.68* 


Integrity (n=31) 


4.46 


4.41 


.40* 


Active listening (n=3 1) 


3.93 


3.55 


.43* 


Creating partnership (n=30) 


3.89 


3.65 


.09 


Stimulation (n=3 1) 


3.92 


3.55 


.31 


Diagnostic skills (n=31) 


3.94 


3.59 


.42* 


Adapting (n=30) 


3.95 


3.64 


.53* 1 


Giving feedback(n=3 1) 


3.93 


3.78 


.43* 



Though the retrospective self-scores are indeed lower than the pretest self-scores, this is for most competencies not 

significant. That is why the pretest scores will be used in the further analysis. 

In Table 10, the pre- and posttest scores are summarized. Again, the score for peers and subordinates is based 
on the average score. A paired samples T-test analysis was executed, to see if differences between pretest and 
posttest are significant (the significant differences are again marked in Table 10). Whenever there is a (-) in the 
table, this indicates that the posttest scores are lower. 



Table 10 Pre- and posttest scores of group 2 and 3 



Group 2 




Pretes 




Posttcst 








S 


sv 


P 


SO 


s 


SV 


P 


SO 


Dealing with stress 


3.82 


3.70 


3.84 


4.17 


3.96 


4.06 


4.22 


3.98 (-) 


Integrity 


4.42 


4.26 


4.46 


4.40 


4.48 


4.66 


4.52 


4.34 (-) 


Active listening 


3.87 


3.58 


3.80 


4.08 


4.03 


4.10 


4.08* 


4.03 (-) 


Creating partnership 


3.90 


3.50 


3.96 


3.98 


4.03 


4.16 


39 ‘ (-) 


3 74 <-) 


Stimulate n 


3.81 


3.64 


3.96 


4.05 


4.00 


4.32* 


4.09 


3.97 (-) 




3.78 


3.56 


3.97 


4.18 


4.05 


4.20 


4.25* 


3.98 (-) 


Adapting 


3.69 


3.36 


3.96 


4.03 


3.95 


4.36* 


3.94 (-) 


3.80 (-) 


Giving feedback 


3.90 


3.56 


4.10 


4.33 


4.15* 


4.32* 


4.33 


4.24 (-) 




3.90 


3.64 


4.00 


4.15 


4.08 


4.27* 


4.17 


4 -0l (-) 




Group 3 __ - — 




Pretest 


Posttest 








S 


SV 


P 


SO 


S 


SV 


P 


SO 


Dealing with stress 


4.09 


4.10 


3.83 


4.04 


4.26 


4.63* 


3.81 (-) 


4.12 


Integrity 


4.34 


4.43 


4.33 


4.48 


4.80* 


4.87 


4.49 


434 (-) 


Active listening 


3.97 


3.53 


3.76 


3.94 


4.26 


4.33* 


3.87 


3-79 0 




3.72 


3.63 


3.71 


4.06 


4.14* 


4.27* 


3.89 


3.89 (-) 


Stimulation 


3.74 


3.43 


3.86 


4.15 


4.00 


4.50* 


4.02 


4.07 (-) 


Diagnostic skills 


3.90 


3.70 


3.92 


4.17 


4.17 


4.47* 


3.99 


3.99 (-) 




3.81 


3.10 


3.73 


4.22 


4.14 


4.17* 


4.01 


3 89 (•) 


Giving feedback 


3.83 


3.67 


3.89 


4.31 


4.46* 


4.63* 


4.17 


4 00 (•) 


Overall score 


3.93 


3.70 


3.88 


4.17 


4.28 


4.48* 


4.03 


4.01 (-) 



Overall score 3.93 I J.7U | — 1-2 

SV^supervisor; P=peers; SO = subordinates; (■) = the posttest lower than the pretest 



As can be seen both group 2 and 3 have higher posttest scores than pretest scores for most behavioral competencies. 
Supervisors of’both groups, have often significantly higher scores at the posttest, as well as the ratees of pup 3 
themselves Peers in group 2, are also significantly more positive at the posttest than at the pretest for two 
competencies (active listening and diagnostic skills). However, subordinates of both groups are more negative 
(though not significantly) at the posttest. 
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Next, group 2 and 3 were compared to see if the gain scores between these two groups are diff ^" t f mm each 
other SiL? group 2 is the training-group, it was expected that the gain scores of this group are higher than 
scores of group 3. The results are summarized in Table 11 (marked coefficient is sigmfican ). 




SV=supervisor; P=peers; SO = subordinates 



A- can be seen in Table 11 most rater sources indicate positive gain scores, with an exception for subordinates and 

significant, and more positive, for group 3. 

Discussion 

ST L c«r«U B wSh .he orher sources. A„ ..he, e.mbin.ri.ns have cxucmcl, low 

C ° TO Nex°t, n ihe 360-degree feedback scores were compared with scores of a ‘more objective’ method, ^“coaching 

sSg^ESSSS&SESSg 

^NexL^f^us^wason d^abUity of 36(Wegree feedback to measure behavioral changes. First, pretest self- 
score"' wirh rarospccl sclf-sc orcs.ro sr^y — ^0'^^ SSSZt 



.. nn , ttest ln a follow-up study interviews with participants will be used to answer the questionwhy subordinates 
Lave P "wer poshest Interesting is also that" the supervisors have the highest gain scores. Though supervisora 

S^A rTtees in group 2 and group 3. have Ae lowest pretest scores for almost all competences, they have Ae 
highest posttest semes for almost all competencies. Again, interviews will be used to try to explamhisre • 

The results further seem to indicate that the gain scores in group 3 are, contranly to what nnghtbe «pected 
hicher Aan Aose of group 2. This is especially the case for self-scores and supervisor-scores. For peers, the results 
„ r ? more mixed witlfsometimes higher scores for group 2 (dealing with stress, active listening, and diagnostic skills 
someLes higher scores for group 3 (integrity, creating partnership, stimulation, adapUng and 

giving feedback) For subordinates Ae results are also mixed, sometimes higher (or better to say: less ne g at tve) for 
giving listening adapting giving feedback and overall score) and sometimes for group 3 (dealing 

However, in ouiy one ce is to differeuce in gn.n scores 

ZT^^rwe. different on seven,, variables. Responden.s in gnrup 2 am mo. often 
maleto on average two year older than respondents in group 3. In addition, respondents in group 3 indicate that 
ftev h“ve inexperience as a senior num. and more experience wirh coaching 1, may be the ease to senior 
Ses mu,7h,ve”me experience in to, new job as a senior, and espeeially wid. coaching, befoul they me able to 
develon their coaching skills. Another explanation may be that the nurses that have been to the training need more 
rime before thev are ably to change their coaching behavior. Note again that at the time of the posttest, the training 
w,?».rL concluded Furiher analysis ol the dam, especially combln.Uon of the gain sco^ with 
background variables, as well as interviews with some of the participants, may clarify this interes ing r 

Conclusions, Recommendations and How This Research Contributes to New Knowledge in HRD 

Organizations are becoming more and more interested in evaluating their training programs at the level of 
behavioral change for which 360-degree feedback is often suggested as a possible method. However, critical 
raiLTons on A?s purpose of 3 60 degree feedback can also be found in literature (see for example McUan etal, 
1993) In this paper the possibility of 360-degree feedback to measure behavioral change was the object t of _smdy^ 
ContLiW to what was expected, Ais study indicated more behavioral change for the control group than for the 
experimental group. Still, this study has provided more insight in the psychometrical properties and the useAlness of 
360-decree feedback However, further experimental research is necessary, especially with a larger number of 
£££, 2, « to cZ l .hi, toy L wilh the possibility lu ruudomly selec. rmees. ro tower die quesn.n 

wheAer 360-degree feedback is indeed useful for training evaluation. 
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Theory-driven Evaluation: An Alternative 



Katie Robertson 
University of Georgia 

While researchers have advanced our ability to understand the effectiveness of Human Resource 
Development (HRD) interventions, practitioners are still in search of an evaluation model capable of 
accounting for the complex relationship that exists between HRD interventions and intended outcomes. 
The purpose of this article is to offer theory-driven evaluation as an alternative to existing HRD evaluation 
models In order to overcome the problems faced by the existing models, this alternative not on y 
establishes a casual link between HRD interventions and outcomes, but also goes further to address 
contextual factors affecting the intervention (Chen &Rossi, 1989). 



Keywords: HRD Evaluation, Theory-driven Evaluation, Review 

The field of Human Resource Development (HRD) has faced the issue of demonstrating that HRD interventions are 
directly contributing to organizational functioning. Therefore, evaluation of HRD interventions has become a 
critical issue in the field (Holton, 1995). It is imperative that HRD evaluators have the tools available to 
demonstrate the utility of HRD efforts. A review of the evaluations models available to HRD practitioners and 
academicians revealed that few models were available. And while recent evaluation strategies have expanded our 
ability to explain the effectiveness of HRD interventions, there still exists a need for an evaluation model capable of 
accounting for the complex relationship that exists between HRD interventions and intended outcomes. To examine 
the common evaluation models and strategies used within the field of HRD, a literature review was conducted. 
Based on the review, theory-driven evaluation is presented as an alternative evaluation approach. 



Evaluation Models in HRD: A Review 

To better understand the problem facing evaluators, a literature review was conducted to explore comnon ^models 
used in HRD evaluation. The review focused on the Academy of Human Resource Development (AHRD) 
conference proceedings for the years 1995-2001. The review revealed that the most common e^luat.onstrateg.es 
in HRD include: Kirkpatrick’s four level typology (Kirkpatrick, 1976 ), Holton s integrated model (Holton 1995) 
the Performance-Learning- Satisfaction evaluation approach (Swanson & McClemon, 1996) Critical Outcome 
Technique (Swanson & Mattson, 1997), and various financial benefit assessments (Brauchle & Koenecke, 1999, 

JaC °Thf mos^saHenf resuh of toe review of the literature was the reliance on Kirkpatrick’s four level evaluation 
model Briefly defined, Kirkpatrick’s hierarchical model has four levels: reaction, learning, behavior, and results. 
Reaction is what the trainee thought of the training intervention. Learning is the quantifiable indication itha 
has taken place. Behavior is the assessment of the transfer of what is learned in the training intervention into 
Dractice In other words, are the skills learned in training being used on the job? Results indicate the impact of the 
Eng on I desired objectives (Kirkpatrick, 1976). While the methodologies for the eva^ion varied, 
Kirkpatrick was the guiding model in many cases (Hastings & Nichols, 1995; Moore, 1999, Nurmi, 1999, 
Ravishankar & Miller, 1997). In a broader literature review. Preskill (1997) found that 77% of journal articles 

pertaining to evaluation models contained Kirkpatrick’s model. . . Th 

While Kirkpatrick’s four level model is a popular basis for research, the model does not go uncnticized. The 
model is often sighted as a mere taxonomy of possible intervention outcomes in need of 

develop the theory (Holton, 1995). Furthermore, in practice the taxonomy is rarely fully implemented (Rothvrell, 
1996) P A survey of evaluation practitioners revealed a narrow range of practices, indicating that Ae method most 
commonly used to evaluate interventions was class evaluations or “smile sheets’’ (Rothwell, 1996). Similar y, 
Dixon (1987) reports that when conducted at all, evaluation usually takes the form of participant reaction. 

While criticism for Kirkpatrick’s model was abundant, alternative evaluation models were scarce. In an attempt 
to strengthen Kirkpatrick’s model beyond a simple taxonomy of results, into one that is able to accoimt for causal 
relationships, Holton (1995) presented an integrated evaluation model. The integrated model adds interveni g 

Copyright © 2002 Katie Robertson 




31-2 

11 



variables that influence the learning and transfer process that have been identified as affecting HRD interventions. 
This model is a step in the right direction, in that it moves away from a simple input/output model. While the 
integrated model presented by Holton is much more inclusive than the simple four level approach that proceeded it, 
several questions are left unanswered. First, this integrated model does not address characteristics of the 
intervention. Cervero (1985) suggests that processes in the design and implementation of the intervention should be 
related to the outcomes. In other words, this model does not account for the type or length of the intervention. 
Second there is little room in the model for intervening variables other than those specified. The model does allow 
for unnamed external events to be added, otherwise, the model is a fixed set of relationships that does not allow 

situationally specific variables to be added. , /mrx « 

A third model identified in the literature review was the Performance-Leaming-Satisfaction (PLS) evaluation 
system (Swanson & McClemon, 1996). The PLS system identifies five major elements, one of which is an 
evaluation model. The model specifies the three domains (performance, learning and satisfaction), and then further 
breaks them down. For example, learning is broken down into knowledge and expertise, performance is divided 
into business results and financial results, and satisfaction is divided in to participant’s satisfaction and sponsor s 
satisfaction. The PLS system also contains an evaluation plan, evaluation tools, an evaluation schedule and an 
evaluation report (Swanson & McClemon, 1996). While this model offers a comprehensive system for approaching 
an evaluation, it is based on the simple input/output concept, which has been criticized in Kirkpatrick's evaluation 

^ A*fourth model that has been presented as an alternative to Kirkpatrick’s four level taxonomy is the Critical 
Outcome Technique (COT) (Swanson & Mattson, 1997). The post-hoc evaluation technique is designed to 
systematically investigate performance outcomes. The purpose of COT is the investigation of intervention outcomes 
through definition, inquiry, verification and valuation of outcomes. This model has the potential to be a practical 
alternative for practitioners who need simple evaluation results, after the intervention has been conducted A pilot 
test of the COT model revealed that the model resulted in information that was useful to intervention stakeholders 
(Mattson, Quartana, & Swanson, 1998). Additional results provided possible refinements to the original model. 

Other evaluation methods identified in the literature review focused on demonstrating die financial benefits ot 
HRD (Brauchle & Koenecke, 1999; Jacobs & Hruby 1996; Swanson, 1998). This information is needed to change 
the notions that HRD is simply an optional activity and the cost of such interventions is greater than the resulting 
benefits (Swanson, 1998). Brauchle and Koenecke (1999) present an overview of contemporary Retum-On- 
Investment (ROI) approaches. The list of possible approaches includes a basic model, a cost'benefit ratio model, 

and benefit -forecasting model. . .. . urlI -, 

While this literature review has focused on evaluation models or strategies appearing in the AHRD conterence 

proceedings, other evaluation models must also be considered. In a recent publication Russ-Eft and Preski 11 (2001) 
comprehensively review eleven evaluation models and approaches. Furthermore, Hilbert, Preskill, and Russ-Eft 
(1997) present a comprehensive review of evaluation models proposed in the last four decades. Included in this 
review are Kirkpatrick’s four level model and Holton’s HRD integrated evaluation model, which are models also 
appearing in the conference proceedings review. However, seven other models not found in the proceedings were 
included in the Hilbert, Preskill, and Russ-Eft review including: Hamblin’s five level model. Training effectiveness 
evaluation system, Binkerhoff six stage model. Input, process, output model, Richey model, Kaufman, Kel or 
Watkin’s five level model, and the Training efficiency and effectiveness model. Dus review concludes that 
evaluation has “focused on a limited number of questions, using only a few tools and methods (p. 145). 

This literature review reveals that evaluators have few alternatives to Kirkpatrick’s simple taxonomy of 
outcomes. Furthermore, the available models do not account for the complex relationship that exists between HRD 
interventions and intended outcomes. While Holton’s (1995) integrated model is moving in the right direction, even 
more is needed to fully account for the complex evaluation context. Preskill (1997) reiterates the concern about : t e 
few evaluation approaches available by noting how rarely “HRD professionals and scholars have tapped other 
disciplines such as public education and the government or military in their development of evaluation the ° n ® s - 
models and approaches” (p.51). The problem then, is that evaluators have few models to choose from and the 
available models do not fully account for the complex HRD intervention context. 

Purpose 

The purpose of this paper is examine the evaluations models available to HRD practitioners. This review reveled 
the dominance of simplistic evaluation models. More specifically, the models identified in the cview can e 
classified as “black box” evaluation models, which focus primarily on input/output relationships, leaving ou 
variables of context. The one-variable black box evaluation may be able to establish a connection between the HRD 
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intervention and outcomes, but it fails to explain the relationship. In order to build causal explanation, causal 
moderators or intervening variables need to be defined. Including such moderators results in a more complete 
understanding of the organizational conditions, individuals, and characteristics of programs that are the most likely 
to produce the desired outcomes (Cervero, 1985). The channels through which the program achieves (or fails to 
achieve) the desired results can then be mapped. By answering the “why” question evaluators will be able to 
contribute to our understanding of how to plan effective HRD interventions (Cervero, Rottet, & Dimmock,1986). 
The problem is that HRD evaluators need evaluation models that open up the black box, resulting in information that 
can contribute to better program design. 



Theoretical Framework 

To address the problem identified above, theory -driven evaluation is presented as an alternative theoretical 
framework.. In order to overcome the problems faced by the existing models, this alternative not only establishes a 
casual link between HRD interventions and outcomes, but also goes further to address contextual factors affecting 
the intervention (Chen &Rossi, 1989). Theory-driven evaluation is a significant contribution to the evaluation 
practitioner because it attempts to move evaluation away from the black box evaluation, and answers questions 
about why the program worked. The remainder of this paper takes an in-depth look at theory -driven evaluation. 



Theory-Driven Evaluation: An Alternative 

Chen and Rossi (1983) developed the concept of theory-driven evaluation as a process capable of incorporating 
existing theoretical knowledge into the evaluation process. Theory -driven evaluation has been a movement of 
critical importance in the last two decades, in that it is a broader perspective of evaluation that has been 
conceptualized in the past (Chen, 1994). Central to theory-driven evaluation is the use of an intervention theory to 
examine the expectations of the intervention (Reynolds, 1998). Intervention theory is essentially a blueprint of how 
the HRD intervention is supposed to work. In other words, it is a “plausible and sensible model of the intervention 
(Bickman 1987 p.5). This model outlines the relationships between the treatment and outcomes. Additionally, 
other factors such as intervening and contextual variables are included (Chen & Rossi, 1989). The completemodel 
should be a broad view of the channels through which the intervention will achieve the desired results. Theory- 
driven evaluation will be farther explained in terms of the components of the intervention theory and the process 
involved in conducting the evaluation. 



Components of an Intervention Theory 

The intervention theory is a complex model that has several key components. These include: the problem, 
intervention strategies, and expected outcomes (Mohr, 1995). In addition to the three main elements of the model, 
other variables, intervening and exogenous, must also be specified (Chen &Rossi, 1983). A more detailed 
discussion of the each of these components follows. 

Three Main Components. When the development process is complete, the intervention theory must include 
specification of three main elements, the problem, intervention strategies, and die expected results (Mohr, 1995). 
First the problem should be defined including behavior to be addressed, where appropriate, and the population tor 
whom the intervention is targeted. In addition to the “what and who” of the problem the context in which the 
problem is located must also be considered. The description of problem context will be useful in explaining or 
interpreting the empirical findings of the evaluation study (Chen & Rossi, 1989). Second, the intervention content, 
or the elements of the intervention that will produce the effect must be outlined. It is important to identify the 
intervention as planned (Chen & Rossi, 1989). It will be shown later that the planned intervention can be very 
different from the implemented intervention. Finally, the intended or expected effects of the intervention must be 
clearlv defined (Reynolds, 1998) The specification of outcomes deserves special attention. Mohr (1995) names the 
comprehensiveness of outcomes identified, as a factor that could increase utilization of evaluation results. It is well 
known that intervention outcomes are not always easy to identify, and the possible outcomes could be niunerous. 
The process whereby the multiple intervention outcomes are specified should be a joint effort between stakeholders 
and the evaluators (Chen & Rossi, 1989). The evaluator has a special role in this process, and that is the 
consideration of unexpected and potentially negative effects of the intervention that stakeholders may overlook. 
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Other Variables in the Model. In addition to the three main elements, intermediate or intervening events must 
also be well specified (Mohr, 1995). Intermediate or intervening variables are factors that link the activities ot the 
intervention with the intended outcomes (Mohr, 1995). For example, if an HRD intervention is designed to reduce 
absenteeism through increased work motivation, both the intervening variable (increased motivation) and the 
outcome variable (decreased absenteeism) must be included in the model, as well as the proposed relationship 
between the two. Not all models will have intervening variables as a part of the intervention theory. However, 
when intervening variables are present they should definitely be included in the model, fhe specification of 
intervening variables is beneficial in that a model that is well specified is more sensitive when tested (Chen & Rossi, 
1983) That is, a model that includes each of the intervening steps can be used to determine exactly how an 
intervention worked or did not work. The intervention theory can then be used to track success, or for revision of 
the intervention as needed. „ . 

There should also be consideration given to what Chen and Rossi (1983) term exogenous variables (p. ). 

Exogenous variables are factors correlated with treatment variables that could also have an impact on the outcome 
variable These variables are included because the outcomes of the complex social interventions are rarely the result 
of the treatment and intervening variables alone (Chen &Rossi, 1983). The effect of the correlation between these 
variables and the treatment can be eliminated by the use of a classical randomized experiment. However, 
randomization is often not possible in applied settings, and the inclusion of the vanables in the intervention theory 

can further explain the intervention contexts and outcomes. 

The Process of Theory-Driven Evaluation 

Now that the concept of intervention theory has been explored, it is important to understand the process 
involved in conducting the evaluation. The process of conducting a theory-driven evaluation can be broken down 
into four steps, including the development of the intervention theory, implementation analysis, selection of relevant 
outcomes, and use of appropriate methodology. It is important to remember that each of the steps in the process is 
linear. In other words, each step relies on the successful completion of the previous step. 

Development of Intervention Theory. The formulation of the intervention theory is an important first step, 
creating the foundation of the evaluation. The theory should be constructed with input from a variety of sources. 
First the evaluator should look to existing literature for research findings that address similar problems, treatments 
or outcomes as the evaluation in question (Chen & Rossi, 1989). Also useful are pre-existing social science theories 
that pertain to the intervention theory being developed. Finally, the evaluator must seek input from intervention 
stakeholders and intervention designers. Often these individuals can provide unique insight into how die 
intervention was designed to address the problem (Chen &Rossi, 1989). During this process each of the intervention 
theory components should be defined. The resulting theory should outline the conceptual framework that is guiding 

the intervention and subsequently will guide the evaluation. 

Implementation Analysis. The implementation of the intervention must also be carefully re viewed (Mohr 1995). 
Implementation analysis is used to insure that the intervention activities are in place to bring about the desired 
outcomes (Mohr, 1995). In other words, it is a procedure used to determine if the HRD intervention is actually 
being conducted as specified during the development of the intervention theory. Chen & Rossi (1983) note that an 
intervention can look very different in theory than it does in practice. To illustrate, the authors cite instances in 
which “evaluators have found that some programs were never implemented, others were placed in the hands of 
inappropriate personnel, and in others implementation effectively negated the program process” (p.303) 

The implementation system must be evaluated because it is a component that necessarily precedes all other 
components of the intervention theory. If the implementation analysis reveals that the intervention is not being 
conducted, or that major discrepancies exist between the intervention theory and the actual intervention, the 
evaluation process should be halted and the intervention theory developed should be revisited and revised. There 'S 
no need to continue to explain relationships between components in the model if the model is known to be incorrect 

These discrepancies in intervention implementation happen because of the complex implementation system that 
support the intervention. These systems consist of administrative rules and regulations, bureaucratic structures, and 
SET who arc responsible for intervention deliver, (Chen & Rossi. 1983). Fortunately, the mterventton 
activities tend to be open and easy to observe. This allows the implementation analysis to be a relatively simple 
Enquiring few of fee evaluation resources (Mohr. 1995). Soon and Seehrcst (1989) have further speetfted 



that the treatment strength should be considered including, intervention specificity, intensity, and duration among 
other factors. 

Determining Relevant Outcomes. As noted earlier, when the intervention theory is being developed, all possible 
outcomes should be identified. This list should include both positive and negative potential outcomes (Chen & 
Rossi 1989). However, it will not be possible to submit all outcomes identified to actual research (Mohr, ). 
Instead the evaluator and stakeholders must collaborate to choose from among all outcomes identified, selecting 
only those most relevant. This is a decision that should be made with input from stakeholders so that the most 
relevant outcomes are selected. Careful selection of outcomes to be tested has the potential to increase utilization o 
evaluation results (Mohr, 1995). 



Selecting Appropriate Methodology. Once established, the intervention theory can be confirmed or denied by 
investigating the linkages between the elements of the theory. Theory -driven evaluation is unique in that it does not 
rely on one type of research methodology (Chen &Rossi, 1989). Instead, a variety of research methods can be used 
to develop and test the intervention theory. In fact, because of the comprehensive nature of heory -driven 
evaluation, multiple methods are often needed to complete the entire process. Quantitative methodologies have 
dominated the discussion of appropriate methodology and will be discussed here. However, it should be noted that 

Qualitative methods hold great promise in the future of theory -driven evaluation. 

There are several techniques that have proven useful for theory verification. Causal modeling is one technique 
used to verify the intervention theory. Causal models can take several forms ncluding, linear structural relations 
(LISREL), confirmatory factor analysis, latent variable structural models, and path analysis (Lipsey & o ard, 
1989) The use of causal modeling techniques requires that each of the components in the model be a variable. In 
other words, the components must vary in such a way that different levels have different values for each component 
fLiDsev &Pollard. 1989). This leads directly to the second requirement, which is that variables must be quantifiable, 
so that the differing levels or values for each component can be identified (Chen & Rossi, 1989). The advantage of 
using causal modeling is that it forces evaluators to get specific on the intervention theory components and the 
relationship between those components. Furthermore, advanced computer programs are available to assist the 
evaluator when testing the model. Lipsey and Pollard (1989) also present a simpler form of causal modeling called 
the “basic two-step” (p.320). The basic two-step narrows the focus of the evaluation to include the first intervening 
variable in the intervention theory, that is, the variable that is expected to change as a direct result of the treatment. 
The second “step” is the more distant variable that is expected to be impacted as a direct result of the changes in the 
first intervening variable. This technique broadens the range of a simple “black box” evaluation considerably by 
considering major mediating variables, but is not as likely to explain complex intervention contexts (Lipsey 

A second methodological approach used in conducting theory-driven evaluation is known as Confirmatory 
Program Evaluation (Reynolds, 1998). This approach is also used to establish causal relationships between 
program intervening variables, and identified outcomes proposed in the program theory, specifically in the post- 
program’ stage and periods of follow-up. While the exact process for conducting a confirmatory program evaluation 
is beyond the scope of this paper, the process is very sinilar to the causal modeling techniques described previously 
(the interested reader is referred to Reynolds, 1998). The process may be applied to experimental quasv 
experimental, or non-experimental data. One benefit of this approach is the ability to strengthen causal inferences 

“in most quastexperimental and non experimental designs” (Reynolds, 1998, p. 207). 

Third Mohr (1995) presents methodology designed to generate measures of effectiveness or adequacy. These 
two measures are based on a comparison between actual results and the counterfactual, the state of the world without 
the intervention. For this comparison to be complete Mohr advocates the use of a control group While the 
identification of this control group is often a difficult task, and by far the strongest limitation of this evaluation 
strategy Mohr does offer many strategies for working with quasiexperimental designs. In addition, regression- 
discontinuity designs, and time series designs have been offered as strategies for evaluation when no control group 
is available. The benefit of this rigorous process is the ability to attribute causality to findings, if and only if designs 
are meticulously executed. 



Conclusions 



Introducing theory-driven evaluation as an alternative to available HRD evaluation models holds great promise. 
Like the Holton (1995) integrated model, theory -driven evaluation goes beyond a simple input/output model to 
consider important contextual factors affecting the intervention. The unique contribution of theory-driven 




31-2 

15 



evaluation is the ability to consider a broader range of influences on intervention success in a situationally specific 
way. Theory-driven evaluation has many advantages, however, it will not be appropriate for every evaluation 
situation. Considering the advantages and disadvantages of the model can help determine when it is an appropriate 
evaluation strategy. 

Advantages 

The advantages of using theory-driven evaluation are numerous. First, the process of developing an 
intervention theory allows for detailed specification of the problem and the target groups. This process may assist 
intervention developers with specifying the reasons for the intervention in question. Furthermore, the intervening or 
causal mechanisms that are designed to address the problem also must be specified. So the way in which the 
intervention leads to expected outcomes must be clearly defined. This approach allows for the success or failure of 
the theory to be established, and also answers the question of why a theory did or did not work. Included in this is 
the distinction between the failure of the intervention to have a causal impact and the failure of the theory to operate 

in the way specified. . 

However, as Finney and Moos (1989) point out, “the greatest benefit of a theoretical foundation for evaluation 
research is a very practical one- the enhanced usefulness of findings” (p.314). More, specifically, findings can help 
make instrumental changes to improve and develop the intervention and also influence the way htervention 
stakeholders conceptualize the problem and the intervention. 

Limitations 

Of course theory -driven evaluation is not appropriate in every evaluation context. The barriers to the use of this 
type of evaluation are quite serious. Bickman (1989) identified multiple barriers to use. First, increased costs can 
be incurred from activities such as, a lengthy planning session, the development of measurement tools and the 
collection of data on the various components of the model. Second, the evaluator must be technically competent in 
the strategies used to implement a theory -driven evaluation approach. In addition, the evaluator is expected to be 
knowledgeable in the problem and treatment literature. This creates the need for evaluators who are technically 
competent in evaluation research techniques as well as HRD. In this situation it is possible for the evaluator to be 
more knowledgeable about the intervention and it’s content than the intervention developer. This could lead to role 
confusion as there are not yet clearly defined role specifications (or limitations) for the theory-driven evaluator. 
Patton (1989) stresses that theory -driven evaluation, which results in generalizations about causal linkages are very 
applicable in some situations, but not all situations. He points out that simpler evaluation models are sometimes 
more appropriate. 

Implications for HRD 

Theory-driven evaluation can contribute to new knowledge in the field of HRD in two ways. First, it may be 
possible to generalize the results of the evaluation to other situations (Chen &Rossi, 1989). In other words, if 
desired outcomes were produced through specified intervening variables, it may be possible to replicate the findings 
in similar intervention situations. Second, knowledge can be generated which contributes to the broader body of 
evaluation knowledge (Chen, 1994). This generation of new knowledge is crucial to the development of program 
evaluation as a field, and in turn to the field of HRD. 

Future Research. The most obvious next step in the incorporation of theory -driven evaluation into the field of 
HRD is the use of the theory in an applied setting. Chen (1994) reiterates this need, calling for more research 
examples in various evaluation situations. The application of this model can be used to test the utility of the various 
methodologies presented. It must be noted however that the possible methods mentioned here are by no means a 
comprehensive set. The list of possible methodologies should be expanded to include qualitative and mixed-method 
designs. A second line of future research is also needed to identify intervening variables common to HRD 
interventions. This list may include known organizational and individual factors that could have an influence on the 
desired outcomes. 

Implications for Practitioners. Theory -driven evaluation has the potential to change to practitioner’s role in 
evaluation quite a bit. There is need for highly trained evaluation practitioners with the skills to carry out the 
rigorous evaluation designs. This involves qualitative skills needed for the development of the intervention theory, 




31-2 



<% 

10 



as well as quantitative skills to test the intervention theory. While the theory-driven evaluation process maybe more 
demanding on practitioners, the result is a comprehensive understanding of the problem, the intervention and 

outcomes produced. 
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Creation and Utilization of Evaluative Information for Organizational Learning 



Youngsook Song 

University of Illinois at Urbana-Champaign 

From the literature review on organizational learning and evaluation, this paper defines the meaning of 
organizational learning and examines organizational conditions enabling the organization to promote 
learning at different levels and in various ways. This paper attempts to explain the linkage between 
organizational learning and evaluation by emphasizing the function of evaluation as a process of creating, 
distributing, and utilizing evaluative information. In addition, the conditions that influence evaluation to 
better function for organizational learning are investigated. 
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Today’s organizations have been experiencing dramatic changes (Marquardt & Berger, 2001). Diverse forces, such 
as rapid development of technology, a more comparative and globalized market, and increased workforce diversity 
influence organizations internally and externally for changes (Dixon, 1992; Schwandt & Marquardt, 2000). To 
survive in this ever-changing society, organizations should be able to adapt and respond to changing conditions and 

learn and transform continuously. - . , • -vtpmai 

Learning in the organization becomes more critical with increased complexity of internal and e 

environments (Sullivan, 1999). Organizations are not any more entities simply controlled and managed depending 
on a linear system model (Hudson, 2000). Inside and outside of the organization, many influences are interacting 
each other and all influences are changing dynamically. It becomes much less predictable to determine 
organizational situations, for instance economic trends, global markets, and intemal/extemal customer needs. 
Today’s organizations coping with escalating changes and complexity cannot help but have a focus on continuous 

organizational learning and development. ... . . ,. , 

In order for the organization to learn, each organizational member should learn first. Although organizational 
learning is more than accumulation of all individual learning, it calls for individual changes in dunking and 
behavioral ways. With respect to the notion of lifelong learning, it is important that every person sh °“ ld “ 
position to keep learning and need to have continuous learning opportunities throughout his or her life (Maehl 
2000) People accustomed to stability and permanence in the workplace may be more likely to face unanticipated 
problems and increasing demands for changes. They could lose even their jobs unless they deal writ these 
unexpected problems and demands effectively. Over the past decade, many organizations have undertaken 
numerous kinds of layoffs in scale with various names of organizational interventions, such as reengineering, merger 
and acquisition, and downsizing, etc. It is essential for the organization as well as for him and her to determine 
whether he or she, as a member of the organization, can leam effectively and efficiently. . 

From the viewpoint of strategic alliance and management, all functions of the organization ought to be 
integrated with each other to achieve organizational goals, and its system and culture should be able tomax.mize all 
employees’ capabilities to contribute to the organization’s success (Gilley & Maycumch, 199 ). mce 
organizational learning is a critically emerging organization goal, organizations need to consider the extent to w ic 
all functions and people in their organizations are eady to complete the goal and enhance it. In this sense, 
evaluation also should be a part of strategic organizational process. Preskill and Toress (1999b) propose that 
evaluation should and can be a means for enhancing learning in organizations. They believe that orgamzat 
learning will occur when organization members are involved in collaborative, communicative, and reflective 

evaluation processes (Preskill & Toress, 2000). . . , 

Evaluation has been performed in numerous ways and on different purposes. In general, it is believed that there 
are mainly three different types of evaluation use based on the purposes they serve, which are instrumenta use 
(decision support and problem solving function), conceptual use (educative function) and symbolic use (political 
function)(see Leviton & Houghes, 1981 for details; Weiss, 1998b). However, as Shulha and Cousins (1997) 
indicate in their literature study on evaluation use, there is the emergence of a conception of evaluation as a 
continuous information dialogue. They discover that there are more frequent occurrences of evaluators and 
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stakeholders sharing responsibility for generating, transmitting, and consuming evaluation information. This 
supports the aspect that there is a growing need to reconsider the roles of evaluation with regard to organizational 
learning and improve the ways in which evaluative information should be created, communicated, and utilized. 

Purpose of the Paper 

Because evaluation is a critical component for organizational learning, it should gain more attention as an effective 
way of creating and sharing information/knowledge and promoting learning. In this regard, the purpose of this 
paper was to examine how evaluation can play a pivotal role to enhance organizational learning by reviewing the 
related literature. More specifically, this review was guided by the following questions: 1) What are important 
organizational conditions to enhance organizational learning? 2) What is the main function of evaluation to enhance 
organizational learning? 3) Under which conditions can evaluation better function to enhance organizational 
learning? 

Organizational Learning 

There have been variations and even confusion in defining organizational learning. Before detecting the meaning of 
organizational learning from the literature, it might be worth mentioning a difference between organizational 
learning and learning organization. Although organizational learning and learning organization are used 
interchangeably in many forms of literature, they are different concepts. Preskill and Torres (1998) describe the 

difference as following. . 

When we talk about learning organizations, we focus on the what, and describe systems, 
principles, and characteristics of organization that learn as collective entities. Organizational 
learning, however, refers to how organizational learning occurs, that is, the skills and processes of 
building and utilizing knowledge. It is particularly concerned with the ways in which learning is 
created and shared, (p. 42) 

Organizational learning has a focus on a process of how organizational members learn at various levels and in 
different ways, whereas learning organization refers to a product as a representation of a desired end ot 
organizational learning (Shwandt & Marquardt, 2000). 

Definitions of Organizational Learning 

Over time various definitions of organizational learning have been offered that help to better understand the 
process of learning in the organization (Jenlink, 1994). As pioneers in the organizational learning field, Argyns and 
Schon introduce the concepts of single, double, and deutero -learning (1996; Argyris, 1998). The organization able to 
do deutro -learning considers as a learning organization, since deutro -learning means conprehensive, systematic, and 
continuous learning. This type of learning may often occur every place in the learning organization ideally. Argyris 
and Schon explain the composite process through which and how the organization responds to external and internal 

influences and develops its own learning system. ...... 

Huber (1990) articulate four constructs related to organizational learning - knowledge acquisition, information 
distribution, information interpretation, and organizational memory (p. 88). He views organizational learning as 
processes of creating, distributing, interpreting, and databasing information/knowledge in multifaceted ways. 
Although he spends much room to explain the elements of “know acquisition”, his main point is to declare how 

effectively the information mechanism within the organization can be developed and work for learning. 

Nonaka and Takeuchi (1995) propose a conceptual framework for managing dynamic aspects of organizational 
knowledge creation processes. They argue that organizational knowledge results from a continuous dialogue 
between tacit and explicit knowledge. According to Nonaka (1994), existing organizational learning theories are 
just related to “internalization”, which is a process of transforming explicit knowledge into tacit knowledge. In 
other words, most organizational learning theories mainly focus on explaining how to obtain 
information/knowledge and why people take actions in certain ways. However, Nonaka and Takeuchi ( 1995 ) 
attempt to expand the notion of organizational learning by suggesting a holistic approach that describes a 
continuous and intensified interaction between different types of knowledge at individual, group and organization 
levels. 

Schwandt and Marquardt (2000) suggest an organizational learning model based on Parsons (1960)’s general 
theory of action. Schwandt (1993) define organizational learning as “a system of actions, actors, symbols, and 
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processes that enables an organization to transform information into valued knowledge which increases its long run 
adaptive capacity” (p. 8). They emphasize the organization system’s ability enabling adapt to its internal and 
external conditions. The system’s ability includes a creative learning capacity through the transformation of 
information into knowledge and a performance-orientation. 

Although the organizational learning theories reviewed here have different focuses and perspectives in 
explaining the concept of organizational learning, three central points can be derived. First, as a major learning 
source, information and knowledge should be generated, distributed, and utilized at various levels in the 
organization. Next, it is critical to develop comprehensive and systematic understanding of interrelations between 
different types of information/knowledge. Lastly, the organizational capacity or ability for learning plays an 
important role to manage and facilitate the organizational learning function and system. 

Conditions for Organizational Learning 

It is significant to identify under which conditions information and knowledge can be better created, distributed, 
and utilized at various levels, and to know what factors need to be considered to promote organizational learning. 
According to Senge (1990), in order to be a learning organization, the organization should embrace and implement 
five disciplines. They are systems thinking, personal master, mental models, building shared vision, and team 
learning. “Systems thinking” refers to a concept for understanding patterns of events and behaviors to know how to 
improve them. This discipline is the most emphasized one among the five disciplines, since he believes that this is 
an essential to integrate all disciplines into each other and to fuse them into a coherent theory and practice of the 
organization. Without systematic understanding, it is evident that there is no motivation as well as channel to see 
how the disciplines interrelate to enhance each other and to lead to organizational learning. 

Watkins (1996) emphasizes organization capacity, motivation, and slack as three triggers for organizational 
learning. According to her, learning occurs when the organization is able to learn (capacity). Learning is critically 
influenced by organizational motivation at various levels (motivation). And, organizational learning takes place 
when financial, technological, or human reserves have spare resources to learn (slack). The organization able to 
learn may be externally and internally well motivated and have abundant learning resources at various levels. 
Marsick and Watkins (1999) address various ways to achieve organizational learning. For instance, work redesign, 
change of reward systems, flatter structure, and team work, etc. are suggested as means of enhancing organizational 
learning. Senge* s view provides useful conceptual principles enabling the organization to be a learning organization 
and gives a thinkable room to link desirable concepts to practice. However, Marsick and Watkins offer more 
tangible guidelines to improve organizational conditions for learning based on their case studies and consulting 
experiences. 

Other studies about organizational learning or learning organization show that organizational culture, 
leadership, formal education programs, and informal learning opportunities critically influence learning in the 
organization. These are in a supportive position for Senge and Watkins’ perspectives overall. Regarding 
organizational culture, both studies of Lancaster (1998) and Argyris (1998) indicate that employees often perform 
unsuccessfully when threatened or embarrassed. Griego (1999) also found that rewards and recognition were 
essential aspects of building a learning organization. According to him, organizational members were more likely to 
define their workplace as a learning organization when they felt their tasks was appreciated, rewarded, not punished 
for mistakes. A learning oriented atmosphere or culture, which includes active organizational efforts to promote as 
well as allow learning with an amount of risk taking, is considered as a key influence on organizational learning. 
Other studies show that how critically organizational learning is associated with some factors such as leaders or 
peers* support (Brooks, 1992; Ellinger, Watkins, & Bostrom 1999; Griego, 1999; Holton & Kaiser, 2000), formal 
training and educational program (Brooks, 1992), and informal or self-directed learning (Confessore & Kops, 1998; 
Durr, Guglielmino, & Guglielmino, 1996; Marsick & Watkins, 1990). 

In short, the main organizational learning conditions required or recommended throughout the literature to 
enhance organizational learning are systematic support, empowering, communication (sharing), and collaboration. 
The organization’s leadership and culture need to be learning flavored and the organization should be able to offer 
various methods and channels for organizational members to be involved in making information and decision and to 
share their knowledge and experiences so that learning occurs across the organization. 



Evaluation 

In order for evaluation to enhance organizational learning, evaluation itself needs to be learning oriented and thus 
effectively create evaluative information. Moreover, information and knowledge generated through evaluation 
should be disseminated at various levels and be used for organizational learning. 
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Evaluative Information for Organizational Learning 

Evaluation is a process of systematically assessing the operation and/or the outcomes of evaluend (a person, a 
thing an idea) against a set of explicit or inplicit criterions (Weiss, 1998a). Through evaluation, evaluators 
generate mainly evaluative information and knowledge of evaluend. Evaluative information means information and 
knowledge with amount of valuing and judging. In effect, evaluation does not intend only to generate evaluative 
information Like Chen (1989)’s claim, in some cases, it should be a critical part of evaluation to understand a 
nature of a program and develop a theory of the program. Patton (1997) asserts that the most important element for 
the success of evaluation is to identify and meet major evaluation clients’ needs. In addition, evaluative information 
can be generated without borrowing a form of evaluation. For instance, critical inquiries, reflection processes, and 
communications with other organizational members help to estimate a phenomenon and result in creating and 
distributing evaluative information or develop an evaluative consensus of the phenomenon (Toress, 1994 ; Preskill, 

1994, 1999b). , . , 

However, evaluation is the most formal, systematic, comprehensive, and professional way to generate, 

distribute, and utilize evaluative information. As already mentioned in the previous section, without a process of 
acquiring’ or producing information and knowledge, organizational learning cannot take place. In terms of 
organizational knowledge construction and intellectual growth, evaluation can play a significant role by creating and 
consuming evaluative information. Evaluative information and knowledge generated and shared through evaluation 
should lead to program development and organizational change (Cronbach and Associates, 1980). It is obvious that 
evaluative information ought to be a critical learning source to foster diverse types and levels of learning and help to 

apply them to visible actions. ..... . ,-r .■ r 

Furthermore evaluation should embrace ample opportunities of creation, distribution, and utilization or 
evaluative information for evaluators and stakeholders. Preskill and Torres (2000) suggest that when organization 
members are involved in the evaluation process that is collaborative and guided by dialogue and reflection, learning 
occurs not only at the individual level but also at the team and organization level. This suggestion means in order 
that evaluative information can te effective within and outside of the evaluation boundary for organizational 
learning, the evaluation process should be learning-centered by offering diverse chances to share and develop 
evaluation participants’ knowledge, skills, and attitudes on evaluend collectively as well as individually. 

Conditions to Generate and Utilize Evaluative Information for Organizational Learning 

In this section, the author discusses that under what conditions evaluative information and knowledge can be 
better created distributed, and utilized to enhance organizational learning. Rogers and Hough (1995) argue that 
evaluation focus methods, and management should reflect realistic assumptions about how organizations work. In a 
similar viewpoint to Rogers and Hough, Jenlink (1994) claims that evaluation needs to take place based on 
understanding how the organization leams and it should function to transform the organization to be a learning 
organization. Rogers and Hough (1995), and Jenlink (1994) stress that it is crucial to take into account the 
organization’ context in terms of its learning infrastructure or “architecture”, so that evaluation functions for 
organizational learning in considering the organization’s nature and being integrated with the place where it is 

^ Th e organization capacity for learning through evaluation is also critical to produce and use evaluative 
information as the organization learning ability is essential when organizational learning occurs (Torress, Preskill, & 
Pionek, 1996). It is not productive and effective to engage in evaluation processes unless the organization is ready 
to use evaluation results and learn from evaluation practice. In this sense, evaluators and stakeholders need to 
determine the extent to which the organization has the readiness for learning through evaluation. 

Moreover Preskill and Torres (1999b) investigate in which ways evaluative inquiry can contribute to the 
maintenance and development of organization (see also Shulha, 2000 ; Rossman & Rallis, 2000). They focus on the 
four elements of organizational infrastructure, which are leadership, forms of communication, systems and 
structures and culture (1999a, 1999b, 2000). According to them, when evaluation is employed as a means of 
accumulation of information for organizational learning and development, these four organizational elements are 
essential to implement evaluation effectively and change the organization to be learning-focused. 

Several other researchers study the factors that influence the evaluation process for organizational learning. As 
highlighting enlightenment of stakeholders and formative evaluation process, Owen and Lambert (1995) claim that 
negotiation of an evaluation plan, heightening awareness, synergetic techniques of reporting, and guidelines and 
ongoing personal level support are important conditions to assist with strategic decision-making committed to 
organizational learning and change. Shulha and Cousins (1997) also provide their insights on what affect creation as 
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well as utilization of evaluation information and findings. They identify predictors for evaluation use and cluster the 
predictors into ten categories such as: quality of evaluation implementation, credibility, relevance, communication, 
user involvement, potential for information processing, clients’ need of information, anticipated degree of program 
change, perceived value of evaluation as a management tool, and contextual characteristics of the decision or policy 
setting (p. 196). Evaluation is required to consider more and more complicated situations and conditions to promote 
learning rather than simply match evaluation clients’ questions with a particular method and approach. And, several 
articles show the ways in which evaluation can bring out successful creation and utilization of evaluative 
information as involving participants in the evaluation process and creating information with collaborative team 
working (Cousins, Donohue, & Bloom, 1996; Fetterman, 1994; Scriven, 1997). .... . , 

The organizational conditions that the author mentioned in the previous organizational learning section can be 
applied to enhance organizational learning through evaluation. In other words, well-customized applications of 
systematic support, empowering, collaboration, and communication to evaluation processes help to better generate, 
distribute, and utilize evaluative information for organizational learning. With a broader and comprehensive 
concern, contextual understanding of the organization and organizational readiness for learning through evaluation 
are considerable elements to maximize creation, distribution, and utilization of evaluative information to lead to 
numerous types and levels of learning across the organization. 

Contribution to HRD Research and Practice 

There have been academic efforts that suggest alternative views to expand the HRD evaluation field (Holton, 1996; 
Kaufman & Keller, 1994), since HRD evaluation theories and practice have been pretty much dominated by 
Kirkpatick (1998)’s' well known four level evaluation model. In this sense, the attempt to link evaluation and 
organizational learning can bring out beneficial possibilities to develop the existing HRD evaluation theories and 
practices. As a more systematic and comprehensive evaluation, learning oriented evaluation approaches give critical 
insights to consider evaluation as a continuous information creation and utilization process for organizational 
learning and development. However, there are not many practical applications and experimental evidences of how 
this discussion can influence action and performance improvement and make actual changes in the organization. 
These approaches seem to be even idealistic or conceptual (Shulha, 2000). 

For further research, it is necessary to know how evaluative information should interrelate and interact with 
different types of information and knowledge to promote organizational learning. More empirical evidences 
supporting the power of evaluation to foster organizational learning need to be mixed with concrete concepts and 
sound principles. Additionally, the linkage between information/knowledge and action/performance through 
evaluation should be more defined and developed. 
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The Effects of Performance Feedback upon Small Businesses: A Literature Review 



Sonji Lee 
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Patricia Bederman Miller 
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This review explores literature related to performance feedback in small businesses. It investigates the 
relationships and significance of performance feedback to the theory and practice of Human Resource 
Development (HRD). It studies research of performance feedback in larger organizations, and it identifies 
concerns about the lack of such research specific to small businesses. This review also exposes the need 
for further study of performance feedback in small businesses and the implications for HRD. 

Keywords: Performance Feedback, Small Businesses, Employee Development 

Businesses, regardless of size, bcation, or type, strive to be successful. This success depends, to a very iargepart, 
upon the degree of employee performance. Practitioners in the field of Human Resource Development (HRD) 
recognize the importance of employees and promote many techniques and methodologies to help assure enhanced 
performance by them. HRD professionals constantly study business environments in their seeking of improvements 
in the performance of personnel. One area of study that is of particular importance in the business arena involves 
the feedback of performance data between employees and their supervisors. 

Problem Statement 

Much research has demonstrated that there is a positive link between improved productivity and die provision of 
effective feedback processes. There are substantial amounts of empirical research regarding the relationship between 
employee feedback and performance in larger organizations, but more research should be conducted in small 

businesses (Westhead & Storey, 1996). - 

It is generally accepted that the performance feedback process and the method of communicating performance 
feedback are key components to effectively develop human resources. However, one of the key issues in HRD is 
the identification of the specific factors that influence the performance of employees. These factors include, but are 
not limited to, communicating; coaching; conducting career workshops and supervisory training; developing and 
implementing effective feedback systems, critical incident programs, and goal setting processes; and increasing 
personal motivation at all levels of an organization. This review explores some of these factors that affect employee 
performance. 

Theoretical Framework 

Various theories support the concepts that feedback to employees provides many benefits, including the 
improvement of their job performances. 



Goal Setting 

The establishment of performance goals is one tool in the process of effective employee performance feedback. 
Setting goals and tracking accomplishments towards these goals are especially effective when mutually developed 
and when the feedback is ongoing and it involves all parties. 

Research has demonstrated that the setting of sensible, clear, short- and long-term goals is a trait or habit of 
successful people and organizations. These people recognize that goals are not easy to achieve and that 
they can fail to reach them. They also know that goals require taking risks. Seldom does anything positive occur 
without the taking of intelligent, well-conceived risks. Further, he risks demand sacrifices, such as time, money, 
reputation, or other resources. But, reaping rewards in the future typically demands making careful investments today. 
The fruits of setting goals and of making changes go to those who prudently take the nsks and make the wise 

investments (Wax, 1997). 
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The theory of goal setting proposes that motivation can be affected positively by utilizing goals. Employees* efforts 
and their focusing of attention, persistence, and selection of work strategies are all improved when well-conceived goals 
are properly presented and acted upon. 

Locke and Latham (1984) state that goal setting is a theory “that helps to explain why other motivational techniques 
or approaches... work** (Locke & Latham, 1984, p, 107). According to them, goal setting is pivotal for the success of 
other motivational approaches, including job enrichment, organizational behavior modification, participation, and 
monetary incentives. These approaches are contingent upon the inclusion of goal setting for their successes (Locke & 
Latham, 1984). 

As an example, in endeavoring to modify employee behavior, the recognition of the presence of goals helps direct 
employee behavior until either the goal is accomplished or else it is rejected or modified. Other researchers of 
motivation agree that goal setting is a well-supported theory of motivation in the workplace (Harris & DeSimone, 1994). 

As a motivational tool, goal setting is extremely valuable. It achieves results, but the results are not automatic. As 
any tool, goal setting must be utilized properly. To be successful, goals must be set systematically and used effectively. 
If done so, goal setting can improve productivity, stimulate creative problem solving, and increase satisfaction, 
confidence, and pride (Locke & Latham, 1984). 

Organizational Development 

Providing feedback to employees is often prompted by the need to initiate and manage change. 

Ivancevich and Matteson (1997) employ the term “organizational development** (OD) to refer to “the process of 
preparing for and managing change** (Ivancevich & Matteson, 1997, p.614). They further refine their definition by 
including long-range planning for improved operations througji better utilization of the resources of the organization. 
They list three sub -objectives of OD: changing values or attitudes, modifying behavior, and inducing change in the 
structure and policy of the organization (Ivancevich & Matteson, 1997). 

Organizational development has been defined by Harris and associates “as a process used to enhance the 
effectiveness of an organization and the well-being of its members through planned interventions** (Harris & DeSimone, 
1994, p. 414). The latter part of this definition deals with intentional and planned interventions, such as introducing and 
implementing employee feedback mechanisms. In some instances, these are called intervention strategies. Regardless 
of terminology, these represent a task or a series of tasks undertaken by an organization (or a segment of an organization) 
where the goals are related to the improvement of the organization. Accordingly, planned intervention or intervention 
strategy is the primary method by which changes are made and improvements occur to the organization (Harris & 
DeSimone, 1994). These changes and improvements are critical components of any successful employee performance 
feedback system. 

Beckhard offers his definition of organizational development as “...an effort [that is] planned, organization-wide, 
and managed from the top to increase organization effectiveness and health through planned interventions in the 
organization’s ‘processes,* using behaviorakscience knowledge** (Beckhard, 1969, p 9). His definition mandates 
planned change through systematic diagnoses, the development of strategic plans aimed at improvements, and the 
mobilization of resources to implement the plan. In addition, it involves the total system. Even though a change may 
focus upon a subpart of an organization, the system to be changed is the total, autonomous entity that is essentially free 
to determine its own plans and future (Beckhard, 1969). 

Leadership 

Successful performance feedback is unlikely to occur without effective leadership from within the organization. 

From Jack Welch, the highly respected and successful former Chairman and Chief Executive of General Electric 
Corporation: “I simply dislike the traits that have come to be associated with ‘managing*.... The word manager has too 
often come to be synonymous with control - cold, uncaring, button-down, passionless. I never associate passion with 
the word manager, and I*ve never seen a leader without it” (Lowe, 1998, p. 72). “I don’t run G.E. I lead G.E. (Lowe, 
1998, p.74).** 

In the Wall Street Journal, Welch is described as “a combination of charismatic preacher, all-knowing judge, 
internal ombudsman and hard driving coach” (Murray, 2000, p. A8). Welch, himself, describes a CEO as being 
equivalent to an orchestra leader. He continues with his principles: “Always meet targets, don’t accept mediocrity, 
embrace change, [and] don’t cling to tradition” (Murray, 2000, p. A8). His dedication to leading and to accepting change 
at General Electric is echoed throughout the article. 
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The first step in changing an organization is to scrutinize every product, service, process, market, customer, 
personnel training and development program, the use of all information, and every other component of the organization 
for the end of its useful life. (Gendron, 1999). 

Culture 

The culture within an organization must accept and actively endorse performance feedback as positive, 
enriching experiences. The alternative is resistance and ineffective change. 

Culture, as found in business organizations, is established during many years and is founded upon many different 
stories and interpretations. As such, it is difficult to change. This resistance to change is, in fact, one of the strengths of a 
business culture. But, correspondingly, it also is a weakness. Culture within a business organization can be changed, but 
doing so demands much time and extensive, careful leadership. The people at the top of the organization must be 
intimately involved and must be seen as being truly committed and involved in the change. But the leadership cannot 
rest solely at the top; it must permeate down through all levels of the organization. Unless everyone actively advocates 
for a particular change, it will be ignored as a passing executive fancy (Nadler et al., 1995). In business, nothing replaces 
solid leadership. 

Motivation 

The appropriate use of feedback can yield substantially improved employee attitudes and motivation. 

In studying motivation, Abraham Maslow’s hierarchy of needs (Maslow, 1970), ranked the needs of people into 
a hierarchy, each level having to be satisfied before motivation will advance to the next level. Maslow ranked the 
needs in his hierarchy according to their apparent importance, placing the most important ones, the physiological 
needs, at the bottom. His theory holds that until these needs are satisfied, the other needs do not play a significant 
role in motivation. Maslow’s hierarchy of needs is relevant to the study of performance feedback, because feedback 
from others effects the highest levels of the hierarchy of attainment satisfaction. 

Hackman and Lawler (1971) found that every higher-order need they studied was positively and significantly 
related to the satisfaction of feedback. However, they found little agreement in the area of which jobs provided the 
highest level of feedback 

Frederick Herzberg (1959) developed the two-factor theory of motivation where hygiene factors were found to 
have little influence on motivation and motivation factors were found to provide motivation. Hygiene factors 
include such things as supervision, working conditions, interpersonal relationships, pay, security, and company 
policies. His motivation factors include achievement, recognition, the work itself, responsibility, advancement, and 
growth. Although the results of the study remain controversial, they are important to consider when examining 
performance feedback systems, because they relate directly to recognition, to the possibility for increased 
responsibility, and to advancement and growth. 

Science-Based 

Rensis Likert’s (1967) work centered on science-based organizational theory emerging from the research 
findings on management and organizational performance and the implications for labor/management relationships. 
Likert’s theories are important to evaluate, because his research considers areas of employee involvement in 
business decisions and the relationship of organizations’ success based on management styles. 

Research Questions, Purpose, Design, and Data Collection 

The purpose of the literature review is to explore the research related to performance feedback in small businesses 
and to consider the implications of performance feedback to HRD theory and practice. The following questions 
were determined: 

1) What is the theoretical framework for researching performance feedback? 

2) How do employees and employers benefit from performance feedback? 

3) What research findings can help the HRD practitioner develop methods and programs to improve relationships 
using performance feedback in small businesses? 

ABI Inform, Psyc Info, ERIC, and EBSCO Host were used to review the scholarly literature. The terms used to 
conduct the searches included Performance Feedback, Performance Evaluation, Employee Development, and Small 
Businesses. 
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Methodology 



This literature review was conducted using a meta-analysis of the current research pertaining to performance 
feedback in small businesses. 

Literature Review 

Job Performance 

According to a study by Minter and Thomas (2000), communication and job performance improves when 
employees are provided with immediate feedback and consultation. Job performance tends to be directly related to 
the quality of relationships between the individuals involved. Providing feedback has been identified as being 
essential for die success of positive working relationships among workers and managers (Deming, 1986; Likert, 
1967). 

Employee Importance 

Arthur Anderson and National Small Business United funded a nationwide research study of small and mid- 
sized businesses entitled “Trends for 2000.” The participating organizations were asked to name their most 
significant challenges to future growth and survival. The top challenge was in finding and retaining qualified 
workers (61%), followed by state and federal regulations (35%) and economic uncertainty (29%). The respondents 
then indicated the methods they used to motivate employees. The answers also included providing a team 
atmosphere (63%) and providing a greater sense of purpose (56%) (Arthur Anderson & National Small Business 
United, 2000). This survey helps validate the importance of employees and employee performance in the world of 
smaller businesses. 

Coaching 

The appraisal of performance appears to be used extensively in business to evaluate and track the outputs of 
employees. From “Leader to Leader”, a publication of the Drucker Foundation and Jossey-Bass (1992), the Paradox 
of Performance Appraisal advocates that the stand-alone performance appraisal is counter-productive and should be 
replaced with a variety of tools and delivery systems that are more effective. One of these is coaching, which, the 
author contends, should replace performance ratings, except when they are required by regulation, union contract, or 
are particularly appropriate in individual circumstances, such as a probationary appraisal for new employees. 
Coaching is not easy to accomplish effectively, but, unless this method is not used, performance appraisals by 
themselves will fail to accomplish their intended roles (McGehee, 2001). 

Encouraging and providing feedback, coaching, and communicating with employees has long been 
acknowledged as beneficial in providing positive working relationships with employees (Deming, 1986; Likert, 
1967). Some previous studies in organizational behavior have attempted to relate, more precisely, how coaching or 
providing feedback affect job performance (Vroom, 1964; Ilgen, Fisher, & Taylor, 1979). 

Minter and Thomas (2000) advocate that the coaching process be employed to aid in the effectiveness of 
performance evaluation systems. Their definition of coaching includes a deep respect for all subordinate individuals 
and trusting that subordinates have the abilities to solve their own problems. If subordinates truly believe that their 
managers are understanding and cooperative, job performance will improve. 

According to Meyer, Kay, and French (1990), coaching should be a day-to-day activity and goal setting, rather 
than criticism, should be used to improve employee performance. They also recommend that evaluations should 
have but one singular purpose. Their study presents the following conclusions: 

1) The evaluation process has too many intended uses and purposes. 

2) These uses and purposes are in conflict with each other. 

3) An evaluation discussion does little to influence future job performance. 

4) Job performance improves when mutually agreed upon goals are established. 

Training 

The training of supervisors to properly use the performance feedback system is extremely important to the HRD 
practitioner. Beyond this, McGehee (2001) advises businesses to train everyone about useful feedback, to provide 
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open systems and structures that encourage feedback, and to use other tools to stimulate commitment to 
improvement efforts throughout every organization. 

Watson Wyatt, a management consulting firm in New York, conducted a study of 1,020 medium to large 
companies in North America and studied 17 of those in depth. It found that career workshops were among those 
activities that led to greater productivity in employees (Comeau-Kirshner, 1998). 

Participation 

Other research has recommended several approaches to improve the performance feedback process. McGregor 
(1990) encouraged that subordinates become active in the evaluation processes by taking responsibility for 
developing a plan for themselves and determining their own potential. McGregor’s research strongly supports the 
use of Management by Objectives (MBO). 

Practitioners recently have published articles relating to the various issues pertaining to performance evaluation. 
Continuous feedback provides employers with very real and important opportunities to inform employees how 
successfully they are doing their jobs, feedback that employees, in general, truly desire. (Laumeyer, 1999). Pardue 
(1999) urges the development and use of performance evaluation systems that are truly participative in nature. 
Pardue (1999) reports that well documented performance appraisal forms help defend various employment 
decisions. Longer-term performance diaries have been recommended as an effective aid to performance raters for 
several decades (Bemardin & Walter, 1977; DeNisi, Robbins, & Cafferty, 1989). 

Levinson (1990) recommends from his research that a detailed critical incident process be used to carefully 
document all incidents of behavior, both good and bad, and done as they occur. These incidents should be discussed 
with employees at the time of the occurrence in order to provide immediate feedback. This will help the employees 
to correct small problems before they develop and expand into larger ones. Levenson purports that these 
documented data then can be used during a subsequent review time to substantiate the details. 



Needs 

Levinson (1990) also suggests that the psychological needs of individuals be taken into consideration during the 
performance evaluation process. As an example, the supervisor should not only be asking questions pertaining to 
the individuals’ personal life goals but should also be inquiring as to what makes them feel good about themselves. 
Levenson’s premise for recommending this is that each person is working towards the attainment of his or her 
individual psychological needs. Therefore, the highest level of individual motivation will arise when a parallel 
relationship exists between the needs of an individual and those required in the job and by the employing 
organization. 

Results and Findings 

The survival of any business, large or small, depends upon the effective performance of tasks by its employees. 
Without feedback, however, employees usually are unable to maintain or improve the desired quality and quantity of 

their work. 

As a result of the comprehensive literature review, it is evident that there is a substantial lack of research data 
on performance feedback that pertains specifically to small businesses. 

All research studies that were explored in this review unanimously endorse the validity of performance 
feedback processes when designed and executed properly. These studies indicate that the benefits accrue to both 
employers and employees. However, given the scant amount of research on performance feedback in the small 
business environment, the assumption that these benefits apply to this group may not be accurate. 



Conclusions and Recommendations 

“Small businesses (firms with fewer than 500 employees) employ 53 percent of the private, non-farm work force, 
contribute 47 percent of all sales in the country, and are responsible for 51 percent of the private gross domestic 
product” (National Small Business United, 1999). 

Small businesses contribute greatly to the economy of our country and contribute trillions of dollars worth of 
economic power annually to the global market. American small business ranks third in the world of economic 
powers, behind only that of the U.S. as a whole and that of Japan. The number of these operations is rapidly 
increasing by about three-quarters of million each year. Most of the new employment in the private sector is from 
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small businesses. The popularly expressed failure rate of small businesses is a myth; in fact, nearly 70 percent of all 
firms beginning in 1985 were still in operation after 10 years (Megginson, Byrd, & Meggison 2000). 

A recent survey by Arthur Anderson and National Small Business United of small and mid-sized businesses 
disclosed that 61 percent of employers expressed that finding and retaining qualified workers was their grea es 

challenge to growth and survival (Arthur Anderson, 2000). 

The. value of effective performance feedback systems is well accepted in HRD and large businesses. It has been 
illustrated that job performance improves when attention is given to employees through feedback, consultation, and 
coaching However, the benefits of such feedback have not been evident in the realm of small businesses. Small 
businesses may not reap the same benefits as do their larger counterparts, mainly due to lack of money, staffing, and 
time to conduct effective performance feedback. A real need exists for additional study to expand the knowledge of 
performance feedback, specifically as it relates to the small business environment. 

Implications for HRD 

The value of effective performance feedback systems is well accepted in the HRD and business worlds. HRD 
practitioners often work in a consulting role to small businesses For HRD professionals t0 °J T t0 

assess feedback processes for small businesses, they first must have appropriate background information related to 
performance feedback. However, the scant amount of research regarding performance feedback processes in small 
businesses tends to limit the effectiveness of the HRD consulting efforts to smaller business entities. Tte pu^oseof 
this review is to demonstrate the opportunities and needs for further research of performance feedback in small 

businesses. 
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